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Abstract 

We propose a model for errors in sung queries, a variant of the hidden Markov model 
(HMM) . This is a solution to the problem of identifying the degree of similarity between 
a (typically error-laden) sung query and a potential target in a database of musical works, 
an important problem in the field of music information retrieval. Similarity metrics are 
a critical component of "query-by-humming" (QBH) applications which search audio and 
multimedia databases for strong matches to oral queries. Our model comprehensively 
expresses the types of error or variation between target and query: cumulative and non- 
cumulative local errors, transposition, tempo and tempo changes, insertions, deletions and 
modulation. The model is not only expressive, but automatically trainable, or able to learn 
and generalize from query examples. We present results of simulations, designed to assess 
the discriminatory potential of the model, and tests with real sung queries, to demonstrate 
relevance to real-world applications. 

1. Introduction 

Many approaches have been proposed for the identification of viable targets for a query in 
a music database. Query-by-humming systems attempt to address the needs of the non- 
expert user, for whom a natural query format - for the purposes of finding a tune, hook 
or melody of unknown providence - is to sing it. Our goal is to demonstrate a unifying 
model, expressive enough to account for the complete range of modifications observed in the 
performance and transcription of sung musical queries. Given a complete model for singer 
error, we can accurately determine the likelihood that, given a particular target (or song in 
a database), the singer would produce some query. These likelihoods offer a useful measure 
of similarity, allowing a query-by-humming (QBH) system to identify strong matches to 
return to the user. 

Given the rate at which new musical works are recorded, and given the size of multimedia 
databases currently deployed, it is generally not feasible to learn a separate model for each 
target in a multimedia database. Similarly, it may not be possible to customize an error 
model for every individual user. As such, a QBH matcher must perform robustly across a 
broad range of songs and singers. We develop a method for training our error model that 
functions well across singers with a broad range of abilities, and successfully generalizes 
to works for which no training examples have been given (see Section 11). Our approach 
(described in Section 9) is an extension of a standard re-estimation algorithm (Baum & 
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Eagon, 1970), and a special case of Expectation Maximization (EM) (Dempster, Laird, & 
Jain, 1977). It is applicable to hidden Markov models (HMM) with the same dependency 
structure, and is demonstrated to be convergent (see Appendix A). 

2. Problem Formulation and Notation 

An assumption of our work is that pitch and lOI adequately represent both the target and 
the query. This limits our approach to monophonic lines, or sequences of non-overlapping 
note events. An event consists of a {Pitch, lOI) duple. lOI is the time difference between 
the onsets of successive notes, and pitch is the MIDI note number^. 

We take as input a note-level abstraction of music. Other systems act on lower-level 
representations of the query. For instance, a frame-based frequency representation is often 
used (Durey, 2001; Mazzoni, 2001). Various methods for the translation of frequency and 
amplitude data into note abstraction exist (Pollastri, 2001; Shifrin, Pardo, Meek, & Birm- 
ingham, 2002). Our group currently uses a transcriber based on the Praat pitch-tracker 
(Boersma, 1993), designed to analyze voice pitch contour. A sample Praat analysis is shown 
in Figure 1. In addition to pitch extraction, the query needs to be segmented, or organized 
into contiguous events (notes). The query transcription process is described in greater de- 
tail in Section 3. Note that these processes are not perfect, and it is likely that error will 
be introduced in the transcription of the query. 



join + local error 



^\> I j^C ^^ 




I iriiB ^aBu.; 



Figure 1: Sample query transcription: from Hey Jude by the Beatles, the word "better" 



Restricting ourselves to this event description of target and query ignores several ele- 
ments of musical style, including dynamics, articulation and timbre, among others. Objec- 
tively and consistently characterizing these features is quite difficult, and as such we have 
little confidence they can be usefully exploited for music retrieval at this point. We acknowl- 
edge, however, the importance of such elements in music query/retrieval systems in general. 
They will likely prove essential in refining or filtering the search space (Birmingham, Pardo, 



1. Musical Instrument Digital Interface (MIDI) has become a standard electronic transmission and storage 
protocol/format for music. MIDI note numbers essentially correspond to the keys of a piano, where 
'middle C corresponds to the integer value 60. 
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Meek, & Shifrin, 2002; Birmingham, Dannenberg, Wakefield, Bartsch, Bykowski, Mazzoni, 
Meek, Mellody, & Rand, 2001). 

We further simphfy the representation by using lOI quantization and by representing 
pitch in terms of pitch class. lOI is quantized to a logarithmic scale, using g = 29 quanti- 
zation levels, within the range 30 msec, to 3840 msec, chosen such that there are precisely 
four gradations between an eighth note and sixteenth note (or quarter note and sixteenth 
note, and so forth.) This representation mirrors conventional notation in Western music, in 
which the alphabet of rhythmic symbols (eighth, quarter, half, etc.) corresponds to a loga- 
rithmic scale on duration (see Figure 2), and has been shown not to substantially influence 
discrimination between potential targets in a database (Pardo & Birmingham, 2002). 




Target: 



II j J rn,n; 



Query, warping 4 

insertion: ^ 



Query, non-warping 
insertion: 



|j ^r3J 



Figure 2: lOI quantization 



Figure 3: Warping and non- war ping in- 
sertions. 



Pitch is translated to pitch class, a representation where all notes are projected into a 
single octave, and are considered in the context of the 12-tone, well-tempered scale. For 
instance, the frequency 453 Hz is "binned" into MIDI note number 70. The corresponding 
pitch class is mod 12 (70) = 10. This addresses two issues: octave errors are quite common 
in some transcriber systems, and pitch class is an effective, if imperfect, musical (Pardo, 
2002) and perceptual (Bartsch & Wakefield, 2001) abstraction. In addition, pitch class 
substantially reduces the model's "alphabet" size. 

In our implementation, pitch bins are not fixed, but vary from query to query. We first 
convert frequency (/) to what might be termed a floating-point MIDI note representation 
(m), using the assumption of equal-temperament (twelve equally spaced semitones per 
octave) and, according to convention, translating a frequency of 440 Hz to 'A' above middle 
'C, or MIDI note value 69: m = 69 + 121og2 ^. When we round to the nearest note 
number, we introduce a rounding error. To minimize this error, we add an offset to the 
note numbers, to account for any overall pitch tendency. For instance, consider a query 
consisting of a sequence of note numbers {48.4, 46.6, 44.4, 43.6}. This query tends to be 
sharp (or flat) by roughly a quarter-tone. By introducing an offset of +0.5 (a quarter-tone), 
we minimize the rounding error, and can thus more closely preserve the contour: without 
the offset, we round to {48, 47, 44, 44}; with the offset, we round to {49, 47, 45, 44}. Since 
transpositional invariance is a central feature of our model, the direction of the offset is 
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irrelevant in this example. Adopting an approach proposed for a QBH "audio front end" 
(Pollastri, 2001), we consider several offsets (O = {0.0, 0.1, ..., 0.9}). Given a sequence 
of note numbers (M = {ml, m2, . . . , m„}), we choose the offset (o G O) such that the 
mean error squared (e = ^i^il™ ° ^°"" '■"^ "-'I ^ jg minimized, and set Pitch[i] equal to 
round(mj + o). 

We choose discrete sets of symbols to represent pitch and duration since, as will be seen, 
a continuous representation would necessitate an unbounded number of states in our model. 
This second event representation is notated: 

ot = {P[t],R[t]) (1) 

for queries (using the mnemonic shorthand observation = (Pitch, Hhythm)). Target events 
are similarly notated: 

d^ = {P[i],R[i]) (2) 

For clarity, we will return to the earlier representation {Pitch[i], IOI[i]) where appropriate. 
The second representation is derived from the first as follows, where 30 and 3840 are the 
lOI values associated with the centers of the shortest and longest bins, and q is the number 
of lOI quantization bins: 

P\i] = mod i2iPitch\i]) (3) 

/ log /Oim -log 30 , ,A 

Ri] = round -^ ^-^ (g - 1 (4 

^^ V log 3840 -log 30 ^^ ^J ^ ' 

The goal of this paper is to present a model for query errors within the scope of this simple 
event representation. We will first outline the relevant error classes, and then present an 
extended Hidden Markov Model accounting for these errors. Taking advantage of certain 
assumptions about the data, we can then efficiently calculate the likelihood of a target 
model generating a query. This provides a means of ranking potential targets in a database 
(denoted {Di, D2, ■ ■ ■}, where Di = {di, ^2, . . .}) given a query (denoted O = {oi, 02, . . .}) 
based on the likelihood the models derived from those targets generated a given query. A 
summary of the notation used in this paper is provided in Appendix B. 

3. Query Transcription 

Translating from an audio query to a sequence of note events is a non-trivial problem. We 
now outline the two primary steps in this translation: frequency analysis and segmentation. 

3.1 Frequency Analysis 

We use the Praat pitch-tracker (Boersma, 1993), an enhanced auto-correlation algorithm de- 
veloped for speech analysis, for this stage. This algorithm identifies multiple auto-correlation 
peaks for each analysis frame, and chooses a path through these peaks that avoids pitch 
jumps and favors high correlation peaks. For a particular frame, no peak need be chosen, 
resulting in gaps in the frequency analysis. In addition, the algorithm returns the auto- 
correlation value at the chosen peak (which we use as a measure of pitch-tracker confidence) , 
and the RMS amplitude by frame (see Figure 1 for instance.) 
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3.2 Segmentation 

A binary classifier is used to decide whether or not each analysis frame contains the begin- 
ning of a new note. The features considered by the classifier are derived from the pitch- 
tracker output. This component is currently in development at the University of Michigan. 
In its present implementation, a five-input, single-layer neural network performs the clas- 
sification. We assign a single pitch to each note segment, based on the weighted average 
pitch by confidence of the frames contained in the segment. An alternative implementation 
is currently being explored, which treats the query analysis as a signal (ideal query) with 
noise, and attempts to uncover the underlying signal using Kalman-filter techniques. 

4. Error Classes 

A query model should be capable of expressing the following musical - or un-musical you 
might argue - transformations, relative to a target: 

1. Insertions and deletions: adding or removing notes from the target, respectively. 
These edits are frequently introduced by transcription tools as well. 

2. Transposition: the query may be sung in a different key or register than the target. 
Essentially, the query might sound "higher" or "lower" than the target. 

3. Tempo: the query may be slower or faster than the target. 

4. Modulation: over the course of a query, the transposition may change. 

5. Tempo change: the singer may speed up or slow down during a query. 

6. Non-cumulative local error: the singer might sing a note off-pitch or with poor rhythm. 

4.1 Edit Errors 

Insertions and deletions in music tend to influence surrounding events. For instance, when 
an insertion is made, the inserted event and its neighbor tend to occupy the temporal space 
of the original note: if an insertion is made and the duration of the neighbors is not modified, 
the underlying rhythmic structure (the beat) is changed. We denote this type of insertion a 
"warping" insertion. For instance, notice the alignment of notes after the warping insertion 
in Figure 3, indicated by the dotted arrows. The inserted notes are circled. For the non- 
warping insertion, the length of the second note is shortened to accommodate the new 
note. 

With respect to pitch, insertions and deletions do not generally infiuence the surrounding 
events. However, previous work assumes this kind of influence: noting that intervallic 
contour tends to be the strongest component in our memory of pitch; one researcher has 
proposed that insertions and deletions could in some cases have a "modulating" effect 
(Lemstrom, 2000), where the edit introduces a pitch offset, so that pitch intervals rather 
than the pitches themselves are maintained. We argue that relative pitch, with respect to 
the query as a whole, should be preserved. Consider the examples in Figure 4. The first 
row of numbers below the staff indicates MIDI note numbers, the second row indicates the 
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intervals in semitones ('u' = up, 'd' = down.) Notice that the intervahic representation is 
preserved in the modulating insertion, while the overall "profile" (and key) of the line is 
maintained in the non-modulating insertion. 



Target: 



Query, modulating 
insertion: 



Query, 

non-modulating 

insertion: 



^-jtttutt: 



60 62 64 65 67 65 64 62 60 
Int:ii2 u2ul ii2 d2 dl d2d2 



eS: 



^S 



g J--.' - 

60 65 67 69 70 72 70 69 67 65 
Iiit:u5 ii2 u2 ul u2 d2 dl d2 d2 




60 65 62 64 65 67 65 64 62 60 
n«:u5 d3u2ulu2d2dld2d2 



Pitcy 



TPitcti, 



lOL 



PitcK 



lOL 



lOL 



PitCflnserl INSERTION 



[Pitch, 



101 



time 



time 



Figure 4: Modulating and non- Figure 5: Insertion of a note event in a 

modulating insertions query 



The effects of these various kinds of insertions and deletions are now formalized, with 
respect to a target consisting of two events {{Pitcha, lOIa) , {Pitchh,IOIb)}, and a query 

{{Pitchc,IOIc), (PitchinsertlOIinsert) {PUchd, lOId)}, where {PitchinsertIO I insert) IS the 

inserted event (see Figure 5). Note that deletion is simply the symmetric operation, so we 
show examples of insertions only: 

• Effects of a warping insertion on lOI: lOIc = I Ola, I Old = 10 lb 

• Effects of a non- warping insertion on lOI: lOIc = 10 la — lOImsert, 10 1^ = lOI^ 



Effects of a modulating insertion on pitch: Pitchc 
Pitch), — Pitcha 

^ V ' 

pitch contour 



Pitcha, Pitchd = Pitchinsert + 



• Effects of a non-modulating insertion on pitch: Pitchc = Pitcha, Pitchy = Pitchy 

In our model, we deal only with non-modulating and non-warping insertions and dele- 
tions explicitly, based on the straightforward musical intuition that insertions and deletions 
tend to operate within a rhythmic and modal context. The other types of edit are rep- 
resented in combination with other error classes. For instance, a modulating insertion is 
simply an insertion combined with a modulation. 

Another motivation for our "musical" definition of edit is transcriber error. In this 
context, we clearly would not expect the onset times or pitches of surrounding events to be 
influenced by a "false hit" insertion or a missed note. The relationships amongst successive 
events must therefore be modified to avoid warping and modulation. Reflecting this bias, 
we use the terms "join" and "elaboration" to refer to deletions and insertions, respectively. 
A system recognizing musical variation (Mongeau & Sankoff, 1990) uses a similar notion of 
insertion and deletion, described as "fragmentation" and "consolidation" respectively. 
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4.2 Transposition and Tempo 

We account for the phenomenon of persons reproducing the same "tune" at different speeds 
and in different registers or keys. Few people have the abiUty to remember and reproduce 
exact pitches (Terhardt & Ward, 1982), an abiUty known as "absolute" or "perfect" pitch. 
As such, transpositional invariance is a desirable feature of any query/retrieval model. The 
effect of transposition is simply to add a certain value to all pitches. Consider for example 
the transposition illustrated in Figure 6, Section a, of Trans = +4. 



Scale= 1.5 
Trans=+4 



^H query 
3 target 



□ overlap 



b) 



Modu=+2 
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Change=L5 



PError=-1 



Figure 6: Error class examples, opening 
notes of Brahms' Cello Sonata 



Figure 7: Tempo increase 



m e-mmor 



Tempo in this context is simply the translation of rhythm, which describes duration 
relationships, into actual time durations. Again, it is difficult to remember and reproduce 
an exact tempo. Moreover, it is very unlikely that two persons would choose the same 
metronome marking, much less unconstrained beat timing, for any piece of music. This is a 
natural "musical" interpretation. We measure tempo relative to the original using a scaling 
factor on rhythmic duration. Thus, if the query is 50% slower than the target, we have a 
scaling value of Scale = 1.5, as shown in Figure 6, Section a. 

In practice, we use quantized tempo scaling and duration values. Note that addition 
in the logarithmic scale is equivalent to multiplication: with quantized lOI values, we 
replace floating point multiplication with integer addition when applying a scaling value. 
For instance, given our quantization bins, a doubling of tempo always corresponds to an 
addition of four: Scale = 2.0 ^^ Scalcguantized = +4. 

4.3 Modulation and Tempo Change 

Throughout a query, the degree of transposition or tempo scaling can change, referred to as 
modulation and tempo change, respectively. Consider a query beginning with the identity 
transposition Trans = and identity tempo scaling Scale = 1, as in Figure 6, Section b. 
When a modulation or tempo change is introduced, it is always with respect to the previous 
transposition and tempo. For instance, on the third note of the example, a modulation of 
Modu = +2 occurs. For the remainder of the query, the transposition is equal to + 2 = +2, 
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from the starting reference transposition of 0. Similarly, the tempo change of Change = 1.5 
on the second note means that all subsequent events occur with a rhythmic scaling of 
1-1.5 = 1.5. 

Consider Figure 7, which plots the apparent tempo scaling in a rendition of "Row, Row, 
Row Your Boat" on a note- by- note basis. While our model considers several interpretations 
of such a rendition, one approach would be to consider a constantly increasing tempo, 
represented by the least-square deviation regression line, with local rhythmic errors (see 
Section 4.4), represented by the note- wise deviations from that line. 

4.4 Local Pitch and lOI Errors 

In addition to the "gross" errors we have discussed thus far, there are frequently local 
errors in pitch and rhythm. These errors are relative to the modifications described above. 
A local pitch error of A*-^) simply adds some value to the "ideal" pitch, where the ideal 
is determined by the relevant target note and the current transposition. A local lOI error 
of A*-^ has a scalar effect (or again, additive in the quantized domain) on the ideal lOI, 
derived from the relevant target note and the current tempo. Figure 6, Section c, shows 
examples of each error. Note that these errors do not propagate to subsequent events, and 
as such are termed non- cumulative or local errors. Transposition and tempo change are 
examples of cumulative error. 

In some cases, there are multiple interpretations for the source of error in a query. 
Consider for instance Figure 8, which shows a specific interpretation of three disagreements 
between a target and query. The second note in the query is treated as a local pitch error 
of -1. The final two notes, which are a semi-tone sharper than expected (+1), are explained 
as a modulation. The error model, described in the next section, considers all possible 
interpretations, for instance considering the possibility that the error in the second note is 
accounted for by two modulations (before and after), and the final two notes by a pair of 
local errors. Depending on our expectation that such things might occur, one or the other 
interpretation might appear more likely. In general, we would prefer to find the most direct 
possible explanations for queries, since an increased likelihood of error in the model can be 
shown to reduce discrimination (see Section 10). 

5. Existing Error Models 

For edits, we assume that overall rhythm is maintained, and make the natural musical 
assumption that edits have a local impact on pitch. Many QBH applications adopt this 
approach to rhythm (Mazzoni, 2001; Meek & Birmingham, 2002; Pauws, 2002; McNab, 
Smith, Bainbridge, & Witten, 1997; McNab, Smith, Witten, Henderson, & Cunningham, 
1996). 

In this study, we are concerned primarily with local and cumulative error. Far less 
is known about this area. This is largely a matter of convenience: a particular musical 
representation will tend to favor one approach over the other. For instance, a pitch- and 
tempo-invariant representation (pitch interval and inter-onset interval ratio) (Shifrin et al., 
2002; Pauws, 2002) establishes a new transposition and tempo context for each note, thus 
introducing the implicit assumption that all errors are cumulative (Pardo &: Birmingham, 
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2002). A study of sung queries (Pollastri, 2001) determined that cumulative error is in fact 
far less common than local error, a conclusion supported by our studies. 

Another approach to the differences in transposition and tempo context is to attempt 
multiple passes over a fixed context model, and evaluate error rigidly within each pass 
by comparing the query to various permutations of the target. Dynamic time-warping 
approaches (Mazzoni, 2001) and non-distributed HMM techniques (Durey, 2001) are well- 
suited to this technique. It is not possible to model, for instance, a modulation, using these 
methods, only local error. A preliminary QBH proposal (Wiggins, Lemstrom, & Meredith, 
2002) recommends a similar approach, grouping together "transposition vectors" connecting 
query and target notes. Such approaches are amenable to extensions supporting cumulative 
error as well, but have not - to our knowledge - been extended in this way. 

An alternative is to normalize the tempo of the query by either automated beat-tracking, 
a difficult problem for short queries, or, more effectively, by giving the querier an audible 
beat to sing along with - a simple enough requirement for users with some musical back- 
ground (Chai, 2001). Again, there is an assumption that the transposition will not change 
during a query, but the beat-tracker can adapt to changing tempi. 

5.1 Alternative Approaches 

We are concerned primarily with sequence based approaches to music retrieval. It is possible 
to relax this assumption somewhat, by translating targets into Markov models where the 
state is simply a characteristic relationship between consecutive notes, allowing for loops 
in the model (Shifrin et al., 2002). Borrowing from the text search world, we can also 
model music as a collection of note n- grams, and apply standard text retrieval algorithms 
(Downie, 1999; Tseng, 1999). In query-by-humming systems, the user is searching for a 
song that "sounds like..." rather than a song that is "about" some short snippet of notes, 
if it makes sense to discuss music in these terms at all^. For this reason, we believe that 
sequence-based methods can more accurately represent music in this context. 





m cu 



Query 



Figure 8: Portion of a query on the 
American National Anthem, 
including examples of modula- 
tion and local pitch error 



Figure 9: Simple HMM, the dishonest 
gambler 



2. Beethoven's Fifth Symphony is a notable exception 
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5.2 Indexing and Optimization Techniques 

It should be noted that as a general model, JCS cannot take advantage of optimizations 
specific to certain specializations. For instance, the cumulative-only version is amenable to a 
relative note representation, a transposition- and tempo- invariant representation, obviating 
the need to compare a query with multiple permutations of the target. 

Existing indexing techniques for string-edit distance metrics - for instance using suf- 
fix trees (Chavez & Navarro, 2002; Gusfield, 1997) and so-called 'wavelet' approximations 
(Kahveci & Singh, 2001) - are appropriate for fc-distance searches, and thus might prove 
useful as a pre-filtering mechanism. However, they are not readily adaptable to more sophis- 
ticated probabilistic matching metrics, and, more importantly, rely on a global alignment 
assumption infeasible for QBH applications. A recent application of dynamic time-warping 
indexing to music retrieval (Zhu & Shasha, 2003) relies on the assumption that the query 
is globally aligned to the target, which is feasible only when the query covers precisely the 
material in the target. 

Other indexing techniques can provide benefits even in the general case. Lexical trees 
are commonly used in speech recognition for large-vocabulary recognition (Huang, Acero, 
& Hon, 2001), and can be adapted to the alignment and metric requirements of JCS us- 
ing similar trie structures. Linear search using sub-optimal heuristics has been applied to 
sequence matching in bio-informatics (Pearson, 1998; Altschul, Madden, Schaffer, Zhang, 
Zhang, Miller, & Lipman, 1997). These approaches are fast for massive databases, and rea- 
sonably accurate even with low similarity distance metrics. Optimal alignment techniques 
- adaptable to arbitrary probabilistic sequence alignment models - are also being devel- 
oped using trie indices (Meek, Patel, & Kasetty, 2003) and parallel architectures (Shpaer, 
Robinson, Yee, Candlin, Mines, & Hunkapiller, 1996). 

We are concerned primarily with determining appropriate metrics for QBH applications. 
Once this has been accomplished, it will be possible to choose and develop appropriate 
optimizations and indexing techniques for large-scale deployment. 

6. Hidden Markov Models 

Hidden Markov Models (HMM) are the basis for our approach. We will begin by describing 
a simple HMM, and then describe the extensions to the model necessary for the current 
task. As suggested by the name, HMMs contain hidden, or unobserved, states. As a simple 
example, consider a dishonest gambler, who is known to occasionally swap a fair dice for 
a loaded dice (with thanks to "Biological Sequence Analysis" (Durbin, Eddy, Krogh, &: 
G.Mitchison, 1998) for the example). Unfortunately, it is impossible to observe (directly) 
which of the dice is being used, since they are visually indistinguishable. For this reason, 
we define two hidden states, a and b, representing the conjectures that the gambler is using 
fair and loaded dice, respectively. Further, we represent our expectation that the gambler 
will switch dice or stay with a dice using a transition diagram, where the transitions have 
associated probabilities (see Figure 9). For instance, the arc from a ^ 6 is labelled 0.1, 
indicating that the probability of the gambler switching from the fair dice to the loaded dice 
after a roll is 0.1, or formally Pr{qt+i = b\qt = a. A) where qt is the current state at time 
interval t, and A is the model. What we can directly observe in this example is the result 
of each roll. While we do not know which dice is being used, we know some distribution 
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over the roll values for each dice (shown at the bottom of Figure 9). These are refereed 
to as observation or emission probability functions, since they describe the probability of 
emitting a particular observation in a state. 

In the music information-retrieval (MIR) context, we have a series of models, each 
representing a possible database target, and wish to determine which is most likely given 
a query, represented by a sequence of pitch and rhythm observations. To draw a parallel 
to our gambler example, we might want to determine whether we are dealing with the 
dishonest gambler described above, or an honest gambler (see Figure 10) who uses only fair 
dice. Given some sequence of observations, or dice rolls, we can determine the likelihood 
that each of the models generated that sequence. 



<S) 



luii 




Figure 10: Simple HMM, the honest Figure 11: A model with skips and null 
gambler states 



6.1 Honest or Dishonest? An example 

The strength of the HMM approach is that it is straightforward to determine the likelihood 
of any observation sequence if the transition probabilities and emission probabilities are 
known. Conceptually, the idea is to consider all possible paths through the model consis- 
tent with the observation sequence (e.g., the observed dice rolls), and take the sum of the 
probabilities of each path given those observations. For instance, the roll sequence {1, 5, 4} 
could be generated by one of four paths in the dishonest gambler model, assuming that the 
dishonest gambler always begins with the fair dice: {{a, a, a}, {a,a,b}, {a,b,a}, {a,b,b}}. 
The probability of the second path, for instance, is equal to the probability of the transitions 
{Pr{a — > a) ■ Pr{a — > 6)=0.9 • 0.1 = 0.09) multiplied by the probabilities of the observations 
given the states in the path (the probability of rolling 1 then 5 with the fair dice, by the 
probability of rolling 4 with the loaded dice: 0.167 • 0.167 ■ 0.5 = 0.0139), which is equal 
to 1.25e-3. To determine the likelihood of the observation sequence given the model, we 
simply take the sum of the probabilities of each path (3.75e-3 + 1.25e-3 -|- 2.78e-5 + 7.50e-4 
= 5.78e-3.) The honest gambler is in effect a fully observable model, since there is only 
a single hidden state. Only one path through this model is possible, and its likelihood is 
therefore a function of the observation probabilities only since the path is deterministic (1.0 
transition probabilities): (0.167)^ = 4.63e — 3. From this, we can conclude that the sequence 
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of rolls 1, 5, 4 is more likely to have been generated by a dishonest gambler, though we 
should note that three rolls do not provide much evidence one way or the other! 

7. Extended HMM 

In the context of our query error model, we account for edit errors (insertions and deletions) 
in the "hidden" portion of the model. Using the notion of state "clusters," we account for 
transposition, modulation, tempo and tempo changes. Fine pitch and rhythm errors are 
accounted for in the observation distribution function. 

7.1 State Definition 

The state definition incorporates three elements: edit type (Edit), transposition (Key) and 
tempo (S'peed). States are notated as follows: 

s^ = {E[x],K[x],S[x]),l <x<n (5) 

If E is the set of all edit types, K is the set of all transpositions, and S is the set of all 
tempi, then the set of all states S is equal to: 

E X K X S (6) 

We now define each of these sets: 

7.1.1 Edit Type 

For the sake of notational clarity, we do not enumerate the edit types in E, but define them 
in terms of symbols that indirectly refer to events in the target sequence, encoding position 
information. There are three types of symbol: 

• Samci: refers to the correspondence between the i note in the target and an event 
in the query. 

• Join\: refers to a "join" of / notes, starting from the z**^ note in the target. In other 
words, a single note in the query replaces / notes in the target. 



i^^ query note elaborating the f^ 
single note in the target is replaced by m notes in the query. 



Elab^,: refers to the j query note elaborating the i target note. In other words, a 



Notice that Samci = Join] = Elabj^, each referring to a one-to-one correspondence between 
target and query notes. In our implementation, Joinj plays all three roles. We generate a 
set of states for a given target consisting of, for each event in the target di : 

• A Samci state. 

• Join states Join\, for 2 < I < L where L is some arbitrary limit on the number of 
events that can be joined. 

• Elaboration states Elab'1^- for 2 < m < M and 1 < j < m, where M is some arbitrary 
limit on the length of elaborations. 
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Why do we have so many states to describe each event in the target? We wish to estabhsh 
a one-to-one correspondence between hidden states and query events, to simplify the im- 
plementation, which is why we introduce multiple states for each elaboration. We choose 
not to implement joins by "skips" through a reduced set of states, or elaborations as null 
states, since as discussed, edits influence our interpretation of the underlying target events. 
Figure 11 illustrates a model with skips and null states. Given our definition of insertion 
and deletion, state si would need separate emission probability tables for each outward arc 
(and thus would be functionally and computationally equivalent to the model we propose). 
As mentioned, we explicitly handle only non-modulating and non- warping insertions 
and deletions (see Section 4.1). As such, when comparing target and query events with 
respect to a join, we generate a longer target note, with the sum duration of the relevant 
target events, and the pitch of the first. Similarly, for an elaboration, we treat a sequence of 
query notes as a single, longer event. Figure 12 shows a portion of the hidden state graph 
relating a target and query through a sequence of hidden states, where the dotted notes are 
examples of each generated note. 



Target: 




- states referencing the 1" target event, 
junction (stiorttiancfnA»p:^ = 




Figure 12: Relationship between states 
and events 



Figure 13: Edit type topology 



Where {Pitch[i],IOI[i]) is the i*^ query note, and {Pitch[t],IOI[t]) the t^^ target note, 
we have the following expected relationships between target and query based on the hidden 
state at time t, qt = {E[t], 0,0), ignoring transposition, tempo and local error for the 
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moment: 



{Pitch[i\,IOI[i\) = {Pitch[t],IOI[t]), if E[t] = Same 

{P^tch[i],J2'jt^^'^ IOI[j]) = {Pitch[t],IOI[t]), if E[t] = Join\ 

{Pitch[i],IOI[i]) = {Pitch[t - J + 1], Yl IOI[k\) if E[t] = Elabf^ 



k=t 



Notice that all events in the elaboration point to a single larger event 

7.1.2 Transposition and Tempo 

In order to account for the various ways in which target and query could be related, we must 
further refine our state definition to include transposition and tempo cluster. The intuition 
here is that the edit type determines the alignment of events between the query with the 
target (see Figure 12 for instance) and the cluster determines the exact relationship between 
those events. 

Using pitch class, there are only twelve possible distinct transpositions, because of the 
modulus-12 relationship to pitch. While any offset will do, we set K = {—5, —4, . . . , +6}. 
We establish limits on how far off target a query can be with respect to tempo, allowing the 
query to be between half- and double- speed. This corresponds to values in the range S = 
{—4, —3, . . . , -1-4} in terms of quantized tempo units (based on the logarithmic quantization 
scale described in Section B). 

7.2 Transition Matrix 

We now describe the transition matrix A, which maps from S x S ^ K. Where qt is the 
state at time t (as defined by the position in the query, or observation sequence), axy is 
equal to the probability Pr{qt = Sx\qt+i = Sy,^), or in other words, the probability of a 
transition from state Sx to state Sy. 

The transition probability is composed of three values, an edit type, modulation and 
tempo change probability: 

O-Xy = 0,^y ■ a^y ■ a^y {6) 

We describe each of these values individually. 

7.2.1 Edit Type Transition 

Most of the edit type transitions have zero probability, as suggested by the state descriptions. 
For instance, Samei states can only precede states pointing to index i -|- 1 in the target. 
Elaboration states are even more restrictive, as they form deterministic chains of the form: 
ElabY\ -^ Elabf\, ^ . . . ^ Elabf-^. This last state can then proceed like Samei, to the 
i + 1 states. Similarly, Join[ states can only proceed to i -|- / states. A sample model 
topology is shown in Figure 13, for M = L = 2. Note that this is a left-right model, in 
which transitions impose a partial ordering on states. 

Based on properties of the target, we can generate these transition probabilities. We 
define -Pjom(i)0 ^s the probability that the i note in the target will be modified by an 
order / join. PEiabih it^) is the probability that the i^^ note in the target will be modified by 
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an order m elaboration. Psameii) has the expected meaning. Since every state has non-zero 
transitions to all states with a particular position in the target, we must insure that: 

L M 

Vi, Psame{i) + ^ Pjoin{i, + X] PElabih m) = I (9) 

1=2 m=2 

This also implies that along non-zero transitions, the probability is entirely determined 
by the second state. For example, the probability of the transition Join\ —> Elah\ ^ is the 
same as for Same^ — > Elab\ i- 

Establishing separate distributions for every index in the target would be problematic. 
For this reason, we need to tie distributions by establishing equivalence classes for edit type 
transitions. Each equivalence class is a context for transitions, where the k^^ edit context 
is denoted C^ . A state that is a member of the k edit context (s^ G C^) shares its 
transition probability function with all other members of that context. Each state Sy has 
an associated A' •* value, which is a classification according to the type (e.g. "Join") and 
degree (e.g. / = 2) of edit type. We define the function P^{/SS^>) as the probability of a 
transition to edit classification A^ •* in edit context k, so that for a transition Sx -^ Sy-. 

a§y = Pf (A(^)) ^ s^eC^ and Sy has edit classification A^^). (10) 

We intentionally leave the definition of context somewhat open. With reference to 
broadly observed trends in queries and their transcription, we suggest these alternatives: 

• The simplest and easiest to train solution is simply to build up tables indicating the 
chances that, in general, a note will be elaborated or joined. Thus, the probabilities 
are independent of the particular event in the target. For instance, our current test 
implementation uses this approach with M = 2 and L = 2, with Psame = 0.95, 
Pjoin = {0.03} and Psiab = {0.02}. 

• Transcribers, in our experience, are more likely to "miss" shorter notes, as are singers 
(consider for instance Figure 1, in which the second and third note are joined.) As 
such, we believe it will be possible to take advantage of contextual information (dura- 
tions of surrounding events) to determine the likelihood of joins and elaborations at 
each point in the target sequence. 

7.2.2 Modulation and Tempo Change 

Modulation and tempo changes are modelled as transitions between clusters. We denote 
the probability of modulating by A^-^' semitones on the i^^ target event as Puoduih ^^^') 
(again defined over the range —5 < A'^ < -|-6). The probability of a tempo change of A*-^ 
quantization units is denoted Pchangeih ^ )> allowing for a halving to doubling of tempo 
at each step (-4 < A^"^) < -^4). 

Again, we need to tie parameters by establishing contexts for transposition (denoted C/^ 
with associated probability function P^) and tempo change (denoted Cf with associated 
probability function Pf). Without restricting the definition of these contexts, we suggest 
the following alternatives, for modulation: 
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• In our current implementation, we simply apply a normal distribution over modulation 
centered at A^^' = 0, assuming that it is most likely a singer will not modulate on 
every note. The distribution is fixed across all events, so there is only one context. 

• We may wish to take advantage of some additional musical context. For instance, we 
have noted that singers are more likely to modulate during a large pitch interval. 

We have observed no clear trend in tempo changes. Again, we simply define a normal 
distribution centered at A'^ = 0. 

7.2.3 Anatomy of a Transition 

In a transition Sx — > Sy (where Sx = {E[x\,K[x\,S[x\)), Sx belongs to three contexts: Cf, 
C^ and C^. The second state is an example of some edit classification A'-^^ so Qxy = 
Pf{A^'). The transition corresponds to a modulation of A*-^ = K[y] — K[x], so a^ = 
P^(A'^'). Finally, the transition contains a tempo change of A^"^-* = S[y] — S[x], so af = 

7.3 Initial State Distribution 

We associate the initial state distribution in the hidden model with a single target event. 
As such, a separate model for each possible starting point must be built. Note, however, 
that we can actually reference a single larger model, and generate different initial state 
distributions for each separate starting-point model, addressing any concerns about the 
memory and time costs of building the models. Essentially, these various "derived" models 
correspond to various alignments of the start of the target with the query. 

The probability of beginning in state Sx is denoted tTx- As with transition probabilities, 
this function is composed of parts for edit type (vrf'), transposition (vr^) and tempo (vrf ). 

Our initial edit distribution (vr^), for an alignment starting with the i^^ event in the 
target, is over only those edit types associated with i: Samci, {Jom'}^2 ^-'^d {Elab'f'i\^^2- 
We tie initial edit probabilities to the edit transition probabilities, such that if Sz directly 
precedes Sx in the hidden-state topology, tt^ = a^. This means that, for instance, the 
probability of a two-note join on the i target event is the same whether or not i happens 
to be the current starting alignment. 

The initial distributions over transposition and tempo are as follows: 

• TiKix)'- the probability of beginning a query in transposition x- Since the overwhelm- 
ing majority of people do not have absolute pitch, we can make no assumption about 
initial transposition, and set t^k{x) = 12 > ~^ 1^ X ^ +6- This distribution could how- 
ever be tailored to individual users' abilities, thus the distributions might be quite 
different between a musician with absolute pitch and a typical user. 

• Tisix)'- the probability of beginning a query at tempo x- Since we are able to re- 
member roughly how fast a song "goes", we currently apply a normal distribution^ 



3. In our experiments, we frequently approximate normal distributions over a discrete domain, using the 
normal density function: y — - — ^ , and then normalize to sum 1 over the function range. 
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over initial tempo, with mean and deviation a = 1.5, again in the quantized tempo 
representation. 

7.4 Emission Function 

Conventionally, a hidden state is said to emit an observation, from some discrete or con- 
tinuous domain. A matrix B maps from S x O ^ 3?, where S is the set of all states, and 
O is the set of all observations. bx{ot) is the probability of emitting an observation ot in 
state Sx {Pr{ot\qt = s^,A)). In our model, it is simpler to view a hidden state as emitting 
observation errors, relative to our expectation about what the pitch class and lOI should 
be based on the edit type, transposition and tempo. 

Equation 7 defines our expectation about the relationship between target and query 
events given edit type. For the hidden state Sx = {E[x\, K[x\, S[x\) , we will represent this 
relationship using the shorthand (P[i],i?[i]) -^ {P\i\,R[t]), mindful of the modifications 
suggested by the edit type. The pitch error is relative to the current transposition: 

A(^) = P[t]-(P[i] + K[x]) (11) 

Similarly, we define an lOI error relative to tempo: 

A.'^^) =R[t]-{R[i]+S[x]) (12) 

To simplify the parameterization of our model, we assume that pitch and lOI error are 
conditionally independent given state. For this reason, we define two emission probability 
functions, for pitch (fc^(ot)) and rhythm (6^(oi)), where bx{ot) = bx{ot) ■ bx{ot)- To avoid 
the need for individual functions for each state, we again establish equivalence classes, such 
that if Sx € C[ , then bx{ot) = P,i{A^'), using the above definition of A^ K Similarly, 
Sx € C/^ implies that bx{ot) = P,j^{A^^'). This means that as a fundamental feature, we 
tie emission probabilities based on the error, reflecting the "meaning" of our states. 

7.5 Alternative View 

For expository purposes, we deflne state as a tuple incorporating edit, transposition and 
tempo information. Before proceeding, we will introduce an alternate view of state, which is 
useful in explaining the dependency structure of our model. In Figure 14. A, the flrst inter- 
pretation is shown. In the hidden states (5), each state is defined by Sj = {E[i], K[i], S\i\) , 
and according to the first-order Markov assumption, the current state depends only on the 
previous state. Observations (O) are assumed to depend only on the hidden state, and are 
definedby Ot = {P[t\,R[t\). 

The second view provides more detail (Figure 14. B). Dependencies among the individual 
components are shown. The E, K and S' hidden chains denote the respective components 
of a hidden state. The edit type {E) depends only on the previous edit type (for a detailed 
illustration of this component, see Figure 13). The transposition [K) depends on both the 
previous transposition and the current edit type, since the degree of modulation and the 
current position in the target infiuence the probability of arriving at some transposition 
level. A pitch observation (P) depends only on the current edit type and the current 
transposition, which tell us which pitch we expect to observe: the "emission" probability is 
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then simply the probability of the resulting error, or discrepancy between what we expect 
and what we see. There is a similar relationship between the edit type {E), tempo (5"), 
and rhythm observation (R). 



E 

S: 
O: 



B. E 





Figure 14: The dependencies in two views of the error model, where shaded circles are 
hidden states (corresponding to the target) and white circles are fully observed 
(corresponding to the query). 



8. Probability of a Query 

In the context of music retrieval, a critical task is the calculation of the likelihood that a 
certain target would generate a query given the model. Using these likelihood values, we 
can rank a series of potential database targets in terms of their relevance to the query. 

Conceptually, the idea is to consider every possible path through the hidden model. 
Each path is represented by a sequence of hidden states Q = {qi, q2, • • • , qx}- This path 
has a probability equal to the product of the transition probabilities of each successive 
pair of states. In addition, there is a certain probability that each path will generate the 
observation sequence O = {oi, 02, . . . , ot} (or, the query.) Thus, the probability of a query 
given the model (denoted A) is: 



Pr(0|A) = Y^ PriO\Q, A)Pr(Q|A) 

VQ 



E 

VQ 



U^Uot) 



t=l 



TT, 
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Yl' 

t=2 



'•qt-iqt 



(13) 



(14) 



Fortunately, there is considerable redundancy in the naive computation of this value. The 
"standard" forward-variable algorithm (Rabiner, 1989) provides a significant reduction in 
complexity. This is a dynamic programming approach, where we inductively calculate the 
likelihood of successively longer suffixes of the query with respect to the model. We define 
a forward variable as follows: 



at{x) = Pr{{oi,02,...,ot},qt = Sx\X) 



(15) 



This is the probability of being in state Sx at time t given all prior observations. We initialize 
the forward variable using the initial state probabilities, and the observation probabilities 
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over the initial observation: 

ai(x) = Pr{{oi},qt = s^jA) = T^xKioi) (16) 

By induction, we can then calculate successive values, based on the probabilities of the 
states in the previous time step: 

n 

at+i{y) = '^at{x)axyby{ot+i) (17) 

x=l 

Finally, the total probability of the model generating the query is the sum of the probabilities 
of ending in each state (where T is the total sequence length): 

n 

Pr{0\X) = Y,(^T{x) (18) 

x=l 

8.1 Complexity Analysis 

Based on the topology of the hidden model, we can calculate the complexity of the forward- 
variable algorithm for this implementation. Since each edit type has non-zero transition 
probabilities for at most L + M — 1 other edit types, this defines a branching factor (b) for 
the forward algorithm. In addition, any model can have at most b\D\ states, where \D\ is 
the length of the target. 

Updating the transposition and tempo probabilities between two edit types (including 
all cluster permutations) requires k = {9 • 12)^ multiplications given the current tempo 
quantization, and the limits on tempo change. Notice that increasing either the allowable 
range for tempo fluctuation, or the resolution of the quantization, results in a super-linear 
increase in time requirements! 

So, at each induction step (for t=l, 2, . . .), we require at most k\D\b'^ multiplications. 
As such, for query length T, the cost is 0{k\D\b'^T). Clearly, controlling the branching 
factor (by limiting the degree of join and elaboration) is critical. A; is a non-trivial scaling 
factor, so we recommend minimizing the number of quantization levels as far as possible 
without overly sacrificing retrieval performance. 

8.2 Optimizations 

While asymptotic improvements in complexity are not possible, certain optimizations have 
proven quite effective, providing over a ten-fold improvement in running times. An alternate 
approach to calculating the probability of a query given the model is to find the probability 
of the most likely (single) path through the model, using the Viterbi algorithm. This is a 
classical dynamic programming approach, which relies on the observation that the optimal 
path must consist of optimal sub-paths. It works by finding the highest probability path to 
every state at time t+1 based on the highest probability path to every state at time t. The 
algorithm is therefore a simple modification of the forward-variable algorithm. Instead of 
taking the sum probability of all paths leading into a state, we simply take the maximum 
probability: 

at+i{y) = max [at{x)axyby{ot+i)] (19) 

x=l 
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A side-effect of tliis change is that all arithmetic operations are multiplications for Viterbi 
(no summations.) As a result, we can affect a large speed-up by switching to a log-space, 
and adding log probabilities rather than multiplying. 

Some other implementation details: 

• The edit topology is quite sparse (see Figure 13), so it is advantageous to identify 
successors for each edit state rather than exhaustively try transitions. 

• There is considerable redundancy in the feed-forward step (for both Viterbi and the 
forward-algorithm) since many state transitions share work. For instance, all tran- 
sitions of the form {E[x],K[x],\j) -^ {E[y],K[y],U) share several components: the 
same edit transition probability, the same modulation probability and the same pitch 
observation probability. By caching the product of those probabilities, we avoid both 
repeated look-ups and repeated multiplications or additions, a non-trivial effect when 
the depth of the nesting is considered over edit type, transposition and tempo. 

8.2.1 Branch and Bound 

Using Viterbi, it is possible to use branch and bound to preemptively prune paths when it 
can be shown that no possible completion can result in a high enough probability. First, 
we should explain what we mean by "high enough" : if only a fixed number (fc) of results 
are required, we reject paths not capable of generating a probability greater or equal to the 
k highest probability observed thus far in the database. How can we determine an upper- 
bound on the probability of a path? We note that each event (or observation) in an optimal 
Viterbi path introduces a factor, which is the product of the observation probability, the edit 
transition probability, and the inter-cluster transition probabilities. Knowing the maximum 
possible value of this factor (/) allows us to predict the minimum "cost" of completing the 
algorithm along a given path. For instance, given a query of length T, and an interim 
probability of at{x), we can guarantee that no possible sequence of observations along this 
path can result in a probability greater than at{x)f We use this last value as a heuristic 
estimate of the eventual probability. 

We can determine / in several ways. Clearly, there is an advantage to minimizing this 
factor, though setting / = 1 is feasible (since no parameter of the model can be greater 
than 1). A simple and preferable alternative is to choose / as the product of the maximum 
transition and maximum emission probabilities: 



/ = max ttxy ■ max I max6„(o) I (20) 

In effect, we are defining the behavior of the ideal query - a model parameterization in 
in which there is no error, modulation or tempo change. 

9. Training 

We need to learn the following parameters for our HMM: 



the probabilities of observing all pitch and rhythm errors (the functions P^^ and P^ 
for all contexts c); 
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the probabilities of modulating and changing tempo by all relevant amounts {P^ and 
Pf ); and, 



the probabilities of transitions to each of the edit types {P^ 



f) 



We fix some parameters in our model. For instance, the initial edit type distributions 
are not explicitly trained, since as described, these are tied to the edit type transition 
function. In addition, we assume a uniform distribution over initial transposition and a 
normal distribution over initial tempo. This is because we see no way of generalizing initial 
distribution data to songs for which we have no training examples. Consider that, for 
instance, the tendency for users to sing "Hey Jude" sharp and fast should not be seen to 
influence their choice of transposition or tempo in "Moon River" . 

We will describe the training procedure in terms of a simple HMM, and then describe 
the extensions required for our model. 

9.1 Training a Simple HMM 

With a fully-observable Markov Model, it is fairly straightforward to learn transition prob- 
abilities: we simply count the number of transitions between each pair of states. While 
we cannot directly count transitions in an HMM, we can use the forward variable and a 
backward variable (defined below) to calculate our expectation that each hidden transition 
occurred, and thus "count" the number of transitions between each pair of states indirectly. 
Until we have parameters for the HMM, we cannot calculate the forward- and backward- 
variables. Thus we pick starting parameters either randomly or based on prior expectations, 
and iteratively re-estimate model parameters. This procedure is known as the Baum- Welch, 
or expectation- maximization algorithm (Baum & Eagon, 1970). 

Consider a simple HMM (denoted A) with a transition matrix A, where axy is the 
probability of the transition from state Sx to state Sy, an observation matrix B where hy{ot) 
is the probability of state Sy emitting observation o^, and an initial state distribution 11 where 
TTx is the probability of beginning in state Sx- Given an observation sequence O = {oi, 02, 
. . . ,, Or}, we again define a forward variable, calculated according to the procedure defined 
in Section 8: 

at{x) = Pr({oi,02,...,Oi},gt = s^|A) (21) 

In addition, we define a backward variable, the probability of being in a state given all 
subsequent observations: 

(3t{x) = Pr{{ot+i,ot+2, ■ ■ ■ ,OT},qt = Sx|A) (22) 

We calculate values for the backward-variable inductively, as with the forward- variable, 
except working back from the final time step T: 

(3t{x) = 1, arbitrarily (23) 

n 

I3t-i{x) = Y,axyby{ot)l3t{y) (24) 

y=l 
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We define an interim variable ^t(x,|/), the probability of being in state Sx at time t and 
state Sy at time t + 1, given all observations: 



Ct{x,y) = Pr{qt = Sx,qt+i = Sy\0,X) (25) 

_ Pr{qt = Sx,qt+i = Sy,0\X) 



Pr{0\X) 
at{x)axyhy{ot+i)l3t+i{y) 
E"=i ELi at{x)axyhy{ot+i)l3t+i{y) 



(26) 

(27) 



Finally, we introduce the variable 7t(a;), the probability of being in state Sx at time t. This 
can be derived from ^((x,y): 

n 

it{x) = Y.Ux,y) (28) 

y=l 

These values can be used to determine the expected probability of transitions and the ex- 
pected probability of observations in each state, and thus can be used to re-estimate model 
parameters. Where the new parameters are denoted 11, A and B, we have: 

TTx = li{x) (29) 

expected number of transitions from Sx — > Sy 
T-1 

^^tix,y) 



t=l 



(30) 



J^7t(3 



expected number of transitions from s^ 
expected number of times in state Sy observing o 



T ( 

sr^\it{y), \iot = o 

~, 1 otherwise 
h{o) = '-^ ^ ^ (31) 

^lAy) 
t=\ 



expected number of times in state Sy 

By iteratively re-estimating the parameter values, we converge to a local maximum (with 
respect to the expectation of a training example) in the parameter space. In practice, the 
procedure stops when the parameter values change by less than some arbitrary amount 
between iterations. 

9.2 Training the Query Error Model 

Our query model has a few key differences to the model outlined above: heavy parameter 
tying, and multiple components for both transitions and observations. The procedure is 
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fundamentally the same, however. Instead of asking "How likely is a transition from s^ -^ 
Sy (or what is axy)T' , we ask, for instance "How likely is a modulation of A*-^) in modulation 
context c (or what is P^ {A^ '))7" To answer this question, we define an interim variable, 

«f-(A«),.)= y /«'<-■ "'• 'fm-m = M-<>^ p^i 

~,f^ I otherwise 

the probability of a modulation of A*-^ in modulation context c between time steps t and 
t + 1. We can now answer the question as follows: 

^f (A^^^) = Jr'f^""'^};^ , (33) 

22t=i 22x=-rAtix,c) 

We use a similar derivation for the other two components of a transition. For the edit type 
function, we have: 



cE(AiE) ._ V^ i^tixy), if £'[y] is an instance of A(^) 



. otherwise 






(34) 

(35) 



and, for the tempo change function, we have: 



■~„ U otherwise 



Pf (A(^)) = "^'fT . (37) 



The emission function re-estimation is more straightforward. For pitch error, we have: 

J- J 7i(y), if observing A^-^^ in this state 

^ „ , „, ~ '*'' '^ I otherwise 
^f (A(^)) = ^ ^T ^ 7^ ; (38) 

and, for rhythm error we have: 

y {jtiv), if observing A^^Mn this state 

. „ , „. ~ ^^ ^ I otherwise 

^c^(A(^)) = ^ ^T ^ 7^ • (39) 

Et=i EsyeCR it{y) 

Again, we are simply "counting" the number of occurrences of each transition type and 
observation error, with the additional feature that many transitions are considered evidence 
for a particular context, and every transition is in turn considered evidence for several 
contexts. A formal derivation of the re-estimation formulae is given in Appendix A. 
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9.3 Starting Parameters 

The components of our model have clear musical meanings, which provide guidance for 
the selection of starting parameters in the training process. We apply normal distributions 
over the error and cluster change parameters, centered about "no error" and "no change", 
respectively. This is based solely on the conjecture (without which the entire MIR exercise 
would be a lost cause) that singers are in general more likely to introduce small errors 
than large ones. Initial edit probabilities can be determined by the hand-labelling of a 
few automatically transcribed queries. It is important to make a good guess at initial 
parameters, because the re-estimation approach only converges to a local maximum. 

10. Analysis 

To maintain generality in our discussion, and draw conclusions not specific to our experi- 
mental data or approach to note representation, it is useful to analyze model entropy with 
respect to cumulative and local error. What influences retrieval performance? From an 
information perspective, entropy provides a clue. Intuitively, the entropy measures our un- 
certainty about what will happen next in the query. Formally, the entropy value of a process 
is the mean amount of information required to predict its outcome. When the entropy is 
higher, we will cast a wider net in retrieval, because our ability to anticipate how the singer 
will err is reduced. 

What happens if we assume cumulative error with respect to pitch when local error is 
in fact the usual case? Consider the following simplified analysis: assume that two notes 
are generated with pitch error distributed according to a normal Gaussian distribution, 
where X is the random variable representing the error on the first note, and Y represents 

the second. Therefore we have: fx{x) = Pr{X = x) = —7k=& ^ and fyiv) = Pr{Y = 

1 ^ 
y) = —T^e 2 . What is the distribution over the error on the interval? If Z is the random 



variable representing the interval error, we have: Z = Y — X. Since fx{x) is symmetrical 
about X = 0, where * is the convolution operator, we have: fz{z) = fx * fviz) = -h=^^^ ^ 
which corresponds to a Gaussian distribution with variance o"^ = 2 (as compared with a 
variance of o"^ = 1 for the local error distribution). Given this analysis, the derivative 
entropy for local error is equal to i(log(27rcr^) + 1) ~ 1.42, and the derivative entropy of the 
corresponding cumulative error is roughly 1.77. The underlying distributions are shown in 
Figure 15. It is a natural intuition that when we account for local error using cumulative 
error - as is implicitly done with intervallic pitch representations - we flatten the error 
distribution. 

While experimental results indicate that local error is most common, sweeping cumu- 
lative error under the rug can also be dangerous, particularly with longer queries. When 
we use local error to account for a sequence of normally distributed cumulative errors rep- 
resented by the random variables Xi, X2, . . . , Xn, the local error (Z) must absorb the sum 
over all previous cumulative errors: Z = ^l^iXi. For example, when a user sings four 
consecutive notes cumulatively sharp one semi-tone, the final note will be, in the local view, 
four semi-tones sharp. If cumulative error is normally distributed with variance o"^, the ex- 
pected distribution on local error after n notes is normally distributed with variance na"^ (a 
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standard result for the summation of Gaussian variables). As such, even a low probability 
of cumulative error can hurt the performance of a purely local model over longer queries. 

The critical observation here is that each simplifying assumption results in the com- 
pounding of error. Unless the underlying error probability distribution corresponds to an 
impulse function (implying that no error is expected), the summation of random variables 
always results in an increase of entropy. Thus, we can view these results as fundamental to 
any retrieval mechanism. 

11. Results 

160 queries were collected from five people - who will be described as subjects A-E, none 
involved in MIR research. Subject A is a professional instrumental musician, and subject 
C has some pre-college musical training, but the remaining subjects have no formal musical 
background. Each subject was asked to sing eight passages from well-known songs. We 
recorded four versions of each passage for each subject, twice with reference only to the 
lyrics of the passage. After these first two attempts, the subjects were allowed to listen to a 
MIDI playback of that passage - transposed to their vocal range - as many times as needed 
to familiarize themselves with the tune, and sang the queries two more times. 

11.1 Training 

JCS can be configured to support only certain kinds of error by controlling the starting 
parameters for training. For instance, if we set the probability of a transposition to zero, 
the re-estimation procedure will maintain this value throughout. 

The results of this training, for three versions of the model over the full set of 160 queries, 
are shown in Figure 16, which indicates the overall parameters learned for each model. For 
all versions, a similar edit distribution results: the probability of no edit is roughly 0.85, the 
probability of consolidation is 0.05 and the probability of fragmentation is 0.1. These values 
are related primarily to the behavior of the underlying note segmentation mechanism. 

In one of the models, both local and cumulative error are considered, labelled "Full" 
in the figure. Constrained versions, with the expected assumptions, are labelled "Local" 
and "Cumulative" respectively. It should be apparent that the full model permits a tighter 
distribution over local error (rhythm error and pitch error) than the simplified local model, 
and a tighter distribution over cumulative error (tempo change and modulation) than the 
simplified cumulative model. 

When JCS has the luxury of considering both cumulative and local error, it converges to 
a state where cumulative error is nonetheless extremely unlikely (with probability 0.94 there 
is no change in tempo at each state, and with probability of 0.93 there is no modulation), 
further evidence (Pollastri, 2001) that local error is indeed the critical component. This 
flexibility however allows us to improve our ability to predict the local errors produced by 
singers, as evidenced by the sharper distribution as compared with the purely local version. 
The practical result is that the full model is able to explain the queries in terms of the 
fewest errors, and converges to a state where the queries have the highest expectation. 
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11.2 Retrieval Performance 

Given the analysis in Section 10, it is interesting to consider the effects on retrieval perfor- 
mance when we assume that only local or only cumulative error occurs. To this end, we 
generated a collection of 10000 synthetic database songs, based on the statistical properties 
(pitch intervals and rhythmic relationships) of a 300 piece collection of MIDI representations 
of popular and classical works. In our experiments, we compare several versions of JCS: 

1. 'Full' model: this version of JCS models both local and cumulative error. 

2. 'Restricted' model: a version of the full model which limits the range of tempo changes 
and modulations (±40% and ±1 semitone respectively). This seems like a reasonable 
approach because training reveals that larger cumulative errors are extremely infre- 
quent. 

3. 'Local' model: only local error is modelled. 

4. 'Cumulative' model: only cumulative error is modelled. 

We first randomly divided our queries into two sets for training the models and testing 
respectively. After training each of the models on the 80 training queries, we evaluated 
retrieval performance on the remaining 80 testing queries. In evaluating performance, we 
consider the rank of the correct target's match score, where the score is determined by 
the probability that each database song would "generate" the query given our error model. 
In case of ties in the match score, we measure the worst-case rank: the correct song is 
counted below all songs with an equivalent score. In addition to the median and mean 
rank, we provide the mean reciprocal rank (MRR): this is a metric used by Text REtrieval 
Conference (Voorhees & Harman, 1997) to measure text retrieval performance. If the ranks 
of the correct song for each query in a test set are ri,r2, . . . ,r„, the MRR is equal to, as 
the name suggests: ^ X^"^^ ^. 
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Full 


Restricted 


Local 


Cumulative 


Simple HMM 


MRR 
Median 

Mean 


0.7778 

1 
490.6 


0.7602 

1 
422.9 


0.7483 

1 
379.5 


0.3093 

68.5 
1861 


0.4450 

5 
1537.5 



Table 1: Overview of rank statistics 

The distribution of ranks is summarized in Figure 18. The rank statistics are shown 
in Table 1. In addition, we provide the Receiver Operating Characteristic (ROC) curves 
for each model (see Figure 17) which indicate the precise tradeoffs between sensitivity and 
specificity, as characterized by the true positive and false positive rates for the model. 
Because there is only one true example in each test, we are concerned primarily with results 
where the false positive rate is very low. For this reason, we present false positives on 
a logarithmic scale. Note that a false positive rate of 10"'^ corresponds to a single false 
positive for the size database we are using. For reference, we indicate the performance of 
a theoretical random model. For comparison, we provide results from another HMM-based 
QBH system, listed as "Simple HMM" (Shifrin et al., 2002), using default parameters. This 
system uses a relative note representation, and has hidden states corresponding to quantized 
instances of the pitch interval and lOI ratio pairs found in the target. 




\ZZ1 Cumulati\e 
I I Simple HMM 



10 



10 10 10 

False positive rate (log scale) 



10" 




Figure 17: ROC curves for various JCS 
implementations *false posi- 
tive rate shown on log scale 



Figure 18: Distribution of ranks 



The cumulative error model and the simple model perform quite poorly in comparison 
with the other approaches, owing to the prevalence of local error in our query collection. We 
see little evidence of the reverse phenomenon: notice that restricting or ignoring cumulative 
error does not have a notable impact on retrieval performance except on the longest queries, 
where there is evidence of degradation in performance using the local approach. This result 
agrees with the basic entropy analysis, which predicts greater difficulty for 'local' approaches 
on longer queries. 
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Full 


Restricted 


Local 


Cumulative 


MRR 

Median 
Mean 


0.8011/0.8911 

1/1 
37/17.9 


0.8320/0.8949 

1/1 
33.6/21.6 


0.8345/0.8676 

1/1 
43.7/21.5 


0.2669/0.6047 

27/1 

150.8/79.0 



Table 2: Results overview for generalization experiments, with baseline results given (re- 
sult/baseline) 



It is informative to examine where JCS fails. We identify two classes of failure: 

• Alignment assumption failure: This is the most common type of error. JCS assumes 
that the entire query is contained in the database. When the segmenter misclassifies 
regions before and after the query proper as notes, this situation arises. JCS must ex- 
plain the entire query in the context of each target, including these margins. JCS does 
however model such added notes within the query, using the elaboration operation. 

• Entropy failure: errors are so prevalent in the query that many target to query map- 
pings appear equally strong. Interestingly, we achieve solid performance in many cases 
where the queries are - subjectively - pretty wildly off the mark. While using a differ- 
ent underlying representation might allow us to extract additional useful information 
from queries, this does not alter the fundamental conclusions drawn about retrieval 
behavior with different approaches to error. 

11.3 Training Generalization 

Because of the redundancy in the query collection - multiple versions of each of the passages, 
and multiple examples of each subject's singing - it is informative to examine retrieval 
performance when the models are trained using examples completely unrelated to the test 
set. We randomly selected four of the eight passages in our study, and two of the five 
subjects, and used these queries for training. The remaining passages, performed by the 
remaining subjects, were used for training. In this way, the two sets do not share either 
singers or passages. 

As a baseline, we provide retrieval results when the model is trained on the test set (see 
Table 2, Figures 19 and 20). 

In these experiments, we see evidence of over-fitting for the more heavily parameterized 
"full" model, attributable in some part to the relatively small training set available for this 
experiment. There is a considerable difference in performance between the baseline and 
generalization results. The restricted model is a good compromise, because the relatively 
small number of parameters limits the risk of over-fitting, while modelling a broad range of 
errors. 



11.4 Run-time Performance 

While various parametrization-specific optimizations are possible, the general implementa- 
tion of JCS has lowest-common-denominator behavior with respect to running time perfor- 
mance. Running on a 1.6GHz Pentium 4 Linux machine, a Java version of JCS performs on 
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Figure 19: Retrieval performance for un- 
related test and training sets 



Figure 20: Baseline performance 



average 15 query to target comparisons per second. Training converges to a local-maximum 
after 16 iterations on average, or just over seven minutes on the same system using 80 
training examples. 

12. Future Work 

Even with the generalizations described in this model, a large number of parameters remain. 
We are currently gathering query data to train the model, as more in-depth evaluations of 
performance on non-synthetic queries will be essential. Various important questions remain 
to be answered, such as the following: 

• What is the effect of query representation, for instance using a conventional note 
representation rather than pitch-class? 

• How can we best tie parameters for training? For efficient training, how many contexts 
can (or should) be established? 

• HMMs are amenable to "frame-based" representations, which would allow us to bypass 
the problematic note-segmentation stage of query transcription. Instead of modelling 
the query as a sequence of discrete note events, it is represented as a sequence of fixed- 
width time-frame analyses. Each state in the target model then has an associated 
distribution over duration - the probability of remaining in the state for some number 
of time- frames. We would like to explore the effectiveness of this approach, particularly 
with regards to the tradeoffs between time and retrieval performance. 

Finally, tests on much larger databases will be necessary. While we believe that meta-data 
in the query process (genre, era, instrumentation) will allow us to restrict searches to a 
subset of a database or library, it is reasonable to assume that a large number of targets 
will be relevant to many searches. 
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13. Conclusion 

We have presented a comprehensive model for singer error when creating a musical query. 
Some of these errors are tempo, pitch (both octave and absolute), melodic contour, and the 
particularly difficult class of joins ("skipped" notes in the query) and elaborations (note 
inserted into the query) . It has been shown that a natural "musical" interpretation of these 
edits helps alleviate the problem of target similarity when edits are observed, noted in an 
earlier QBH feasibility study (Sorsa, 2001). 

Our experimental results show that the model is effective in handling these errors. By 
this, we mean that we can identify when these errors occur and can account for them in the 
retrieval task when necessary. The experimental results indicate good retrieval performance 
when these errors occur in queries. Moreover, our model is able to separately model cumu- 
lative error and local error, or a combination of both. Generally, music-information retrieval 
researchers have been divided on the importance of these errors. We have been able to both 
theoretically and experimentally show that local error is more important. That is, ignor- 
ing cumulative error, in general, has a less deleterious effect on retrieval performance than 
ignoring local error. In addition, modelling both errors is may give improved performance, 
but at a computation cost. 

Finally, our model demonstrates the most parsimonious way to model errors. We use 
the idea of a distributed state representation in our HMM to express the difference between 
a query and a target. This allows us to keep the model compact, improving both runtime 
efficiency and retrieval precision. 
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Appendix A. Deriving Re-estimation Formulae 

The reestimation procedure converges to a critical point in the parameter space with respect 
to likelihood. Baum defines an auxiliary function q, where A' represents the "current" model 
parameter values, and we are attempting to iteratively reestimate A: 



q(A', \)=Y, PiO, Ql A') log PiO, Q\X) (40) 
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By maximizing this function, we maximize P{0\X) because: 

q(A', A) > q(A', A') ^ P(0|A) > P(0|A') 



(41) 



We will now derive this implication. Notice that P{0\X) = X^n -P(0, Q|A), or the sum of 
the probabilities of all possible paths through the model. There are a finite number N of 
paths. Where Qi is the i^^ path, pi = P{0,Qi\X'), J2 is shorthand for J2i=i ^^^ qi = 
P{0,Qi\X), we can rewrite the implication: 



^Pi logqi > ^pi logpi ^J2Qi> J2p^ 
The derivation is as follows: 

^ Pi log gi > ^ Pi log Pi 



> 



Pi 



Since x — 1 > logx, we can deduce from Equation 44 the following inequality: 



Y,P^i^-l)>^pJog^>0 



P'. 



Pi 



^Qi-^Pi > "^ Pi ^Og Qi-"^ Pi log 



Pi 



(42) 

(43) 
(44) 

(45) 
(46) 



We know from the original implicant that YlPi^^SQi ^ S Pi log Pi, so the right-hand side 
of Equation 46 is known to be non-negative. Therefore, the left-hand side of the inequality 
must also be non-negative. Since P{0\X) = Y^Qi and P{0\X') = "^Pi, it is clear that 
P{0\X) - P{0\X') > and therefore P{0\X) > P{0\X'). 

To derive our reestimation formulae, we first decompose q(A'[A) into a sum of auxiliary 

functions of the form: 

N 

f{y) = Y,Wj\ogyj, (47) 

i=i 

Where we are constrained by y^ > and g{y) = Yli ?/« = 1 (a discrete probability function), 
the auxiliary functions can then be individually maximized using Lagrange multipliers. In 
general, where Vg{y) is the gradient of the function fl'(y), we know that extremes values of 
the function /(y) subject to constraint g{y) are solutions to the equation: Vg{y) = A;V/(y) 
where k is some constant: 



/ 
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from the constraint 2, "^i = 1 



(48) 

(49) 
(50) 



i=l 
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To reduce q to this form, we rewrite: 

T 

P{0,Q\\) = T,,,J[a,,_,,,h,,{ot) (51) 

Incorporating the various components of transition and emission probabihties, we have: 

T 

P(0,Q|A) = vr,, n«ti..<-...<-...<(°*)&S(o*) (52) 

i=2 

Finahy, we incorporate the notion of context {i^j^k^l^m) and amount/error/symbol (A), 
determined according to the procedure defined in Section 7: 

P(0,Q1A) = /^^(A(^))n/^^(A(^))F,^(AW)Pf (A(^))P,^(A(^))P^(A(^)) (53) 

t=2 

Converting to a log-scale, we have: 

X:iog/^^(A(^)) + X:iog^^(A(^))+ (54) 

f:iogPf(A(^)) + ^logP,^(A(^)) + f^logP«(A(«)) 

t=2 t=2 t=2 

Using this derivation, we rewrite q: 

q(A', A) = J^qf (A',af) + J^qf (A',af ) + (55) 



j; qf (A', af ) + j; qf (A', af ) + j; q^(A', a^ 
where 



x . 
Vfc Ml Vm 



qf (A',af) = J^ 5^ P(0,gt e C!^,qt instance of A^^^jA') log /^^(A^^)) (56) 

T 
qf (A',af ) = 2^ ^ P{0,qt-i - <Zt instance of AW,^* G Cf |A') logPf (A(^)) (57) 
t=2 VA(^) 
T 
qf(A',af)=J] J] P(0,g,_i^(?,instanceofA(^),gtGCf|A')logPfc^(A(^)) (58) 
t=2 VA(S) 
T 
qf (A',bf) =^P{0,qt -^ ot instance of A'^^\qt G Cf |A') logP;^(A(-P)) (59) 
t=i 
T 

q^(A',b^) = ^P(0,gt ^ Ot instance of A^^),^* G C^|A') logP^(A(«)) (60) 

i=l 

Using the result in Equation 50, it is then trivial to derive the reestimation equations 
described in Section 9. 

90 



A Comprehensive Trainable Error Model for Sung Music Queries 



Appendix B. Notation 

We will now outline the notation used in describing the error model: 



Notation 


Description 






{Pitchlx],IOIlx]} 


x*^^ note event 


{-PN.-RN) 


X note event, quantized 


Of and di 


t observation (query note) and 
i target event (database note) respectively 


A(^) ^-1 


a pitch error, one semi-tone fiat 


a(^^) ^+1 


a rhythm error, one quantization unit too long 


s, ^{E[x],K[x],S[x]) 
- {5amei,+2, -3) 


x^'^ HMM hidden state 

E[x] — Same\: Edit type, replacement of first target note 

K[x] — +2: transposition (Key), 2 semi-tones sharp 

S[x] — —3: tempo Scaling, 3 units faster 


^^y = ^Sv ■ ^^v ■ ^^v 


probability of a transition from hidden state s^ — » Sy 


af^ - Pf (A^^)) 


edit transition probability, with edit symbol A*^ ' in context C^ 


a-,^Pf(AC-)) 


probability of a modulation A^ ^ in context C- 


-'^y = P^(^^'^^) 


probability of a tempo change A'^*^-' in context C^ 


A(-^) e {Samei} U {Join\}{'_^ 


set of all possible edit symbols, 
where L and M are the "order" of the edit topology 


at(x) ^ P({oi,02, . . ■ ,ot},qt ^ s^|A) 


forward-variable, the probability of ending in state t at time t 
given model A 


(3t(x) ^ P({ot + i,Ot + 2> ■ . . .OT},qt ^ Sa;|A) 


backward-variable, the probability of beginning in state x at time t 
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